To deal with too many parameters and too much computation in Deep Neural Networks （DNNs）， an unlabeled neural network pruning algorithm based on Bayesian optimization was proposed. Firstly， based on a global pruning strategy， the sub-optimal compression ratio of the model caused by layer-by-layer pruning was avoided effectively. Secondly， the pruning process was independent on the labels of data samples， and the compression ratios of all layers were optimized by minimizing the distance between the output features of pruning and baseline networks. Finally， the Bayesian optimization algorithm was adopted to find the optimal compression ratio of each layer， thereby improving the efficiency and accuracy of sub-network search. Experimental results show that when compressing VGG-16 network by the proposed algorithm on CIFAR-10 dataset， the parameter compression ratio is 85.32%， and the Floating Point of Operations （FLOPS） compression ratio is 69.20% with only 0.43% accuracy loss. Therefore， the DNN model can be compressed effectively by the proposed algorithm， and the compressed model can still maintain good accuracy.

Aiming at the premature convergence problem of standard Particle Swarm Optimization (PSO) in solving complex optimization problem, a new search PSO algorithm based on gradient descent method was proposed. Firstly, when the global extremum exceeds the preset maximum number of unchanged iterations, the global extremum was judged to be in the extreme trap. Then, the gradient descent method was used to proceed twice search, a tabu area was constituted with the center of optimal extremum point and the radius of specific length to prevent particles repeatedly search the same area. Finally, new particles were generated based on the population diversity criteria to replace the particles that would be eliminated. The twice search algorithm and other four improved algorithms were applied to the optimization of four typical test functions. The simulation results show that the convergence accuracy of the twice search particle swarm algorithm is higher up to 10 orders of magnitude, the convergence speed is faster and it is easier to find the global optimal solution.